This week I finished an upgrade from Elixir 1.10 to 1.11.3 and more notably Phoenix 1.4 to 1.5.7. To get a sense of the risk I decided to upgrade Elixir first and keep the bulk of our dependencies as-is to measure what I was up against.
Thankfully the Elixir upgrade, aside from a handful of compiler warnings, wasn't worth writing about. The downside of this easy upgrade was overconfidence which revealed itself as an inability to estimate the effort required to move forward with the latest Elixir, Phoenix, Ecto and Absinthe dependencies.
I've done platform upgrades like this in the past and usually failed to share any lessons learned with the wider community. This time around I decided to take the time and document my adventure to help anyone else who might find themselves in a similar situation.
I'll be doing a brain dump of the most memorable problems I faced with enough detail to unblock others who might feel stuck like I did at times. I would have struggled much more if it wasn't for all the wonderful blog posts, issues and contributions in our community so to all who paid it forward "Thank You!".
PubSub 2.0
The biggest breaking change was the PubSub upgrade from 1.0 to 2.0. I would guess most had no trouble because the guide was simple and straight forward. Just bump the dependencies, update the config and tweak the application.ex but for those of us who used the private api Phoenix.PubSub.Local.list ...well that's when things got interesting.
At first glance PubSub 2.0 had a clear replacement for this functionality in Phoenix.Tracker. The trouble was that Phoenix.Tracker centers around distributed presence tracking and what I needed was a metric about connected users for the local node to restore the Kubernetes autoscaling solution my team put together last year.
I threw together a simple demo app that shows the full solution for anyone who might be in a similar situation. TL;DR I used the Registry to do the heavy lifting.
def handle_info(:after_join, %{assigns: %{session_id: session_id}} = socket) do
{:ok, _} = Registry.register(Subpub.Tracker.Registry, "user_sockets", session_id)
{:noreply, socket}
end
Now I could use the Registry to surface this metric about the local node. Thankfully this was a drop in replacement for the behavior my team used in PubSub 1.0 and as a result the autoscaling feature was back in action with complete feature parity.
Registry.lookup(Subpub.Tracker.Registry, "user_sockets")
handle_out/3 is undefined or private
Next was another PubSub related issue that showed itself at runtime when using Broadcast. Specifically, broadcasting to a Phoenix Channel without Phoenix. The biggest hurdle was the runtime error itself `handle_out/3 is undefined or private` which was not immediately obvious.
def broadcast(topic, event, payload) do
message = %Phoenix.Socket.Broadcast{
event: event,
payload: payload,
topic: topic
}
Phoenix.PubSub.direct_broadcast(Node.self(), My.PubSub, topic, message)
end
When I started searching around for clues on this I was lucky enough to find a forum post where Chris explains the problem in detail. Luckily the solution was simple enough, starting with the move away from `direct_broadcast` in favor of `local_broadcast`.
def broadcast(topic, event, payload) do
message = %Phoenix.Socket.Broadcast{
event: event,
payload: payload,
topic: topic
}
Phoenix.PubSub.local_broadcast(My.PubSub, topic, message)
end
Finally, I added a `handle_info` function to any channel that was used in this way to accomplish the push.
def handle_info(%{topic: _, event: event, payload: payload}, socket) do
push(socket, event, payload)
{:noreply, socket}
end
Absinthe Ecto
Anyone familiar with open source knows that a young ecosystem will see some amount of churn and Elixir is no different. As part of the upgrade I found a handful of places the assoc helper from absinthe_ecto was used for `belongs_to` and `has_many` relationships. As part of the Phoenix upgrade, version compatibility became a problem with this hex library because of the bump to ecto_sql 3.5.3.
object :post do
field :user, :user, resolve: assoc(:user)
end
Because the library is deprecated I removed it and pulled in the source code for assoc to move forward. Side note: I did spend a few minutes looking at dataloader but decided to pull that in another day when I'm battling n+1 issues.
Ecto
In many ways the upgrade to Ecto 3.5 was painless, but I did see a few instances that failed to compile.
schema "posts" do
field :comments, {:array, BrokenSchema.Comment}, virtual: true
end
I did find a useful github issue that José Valim commented on.
The error message is correct, as Comment is not an Ecto.Type. I am actually surprised to how it worked before...
To workaround this I simply flipped to the `any` Ecto.Type and the compiler was happy.
schema "posts" do
field :comments, {:array, any}, virtual: true
end
JSON Serialization
The most frustrating part of the upgrade was the move from Poison to Jason for serialization. Because so much of the Phoenix app is consumed by another client I ran into several stumbling blocks worth mentioning.
The first problem was a runtime error `(FunctionClauseError) no function clause matching in Jason.Encoder` when a field on a given Ecto schema didn't have the correct type during serialization. The biggest challenge with this error is that it failed to offer much information about how to resolve it. I did find a github issue from Oct 2020 that got the ball rolling. And not long after a pull request was merged to give these warnings at compile time.
The only trouble with this was that no public version was published so I pulled down the source code for the latest encoder and used it locally to leverage the compiler. You could also reference the github commit SHA if you prefer. Either way, having these errors at compile time was a game changer so huge thanks to the team for recognizing this and working to resolve it quickly.
The next problem was a runtime error `cannot encode association :comments from Post to JSON because the association was not loaded`. The simple workaround for this problem was to add the association to the `except` list for `derive` but the real problem was how often this had to be done and how painful it was to find all of the edge cases.
@derive {Jason.Encoder, except: [:__meta__, :comments]}
In sharing about this, I hope someone from the community can suggest a better solution. At worst, I found all the places I sorely needed controller tests to catch regressions like this.
The last problem was that previously all keys would be serialized for any embedded_schema even if the key wasn't explicitly declared as a field. This was trouble because more times than I'd like to admit, `Map.put` was used to insert a new key and value that wasn't declared meaning that at runtime those values wouldn't show up in the json.
embedded_schema do
field :title
end
def make_fun(data) do
data |> Map.put(:author, user)
end
Thankfully the solution was easy enough. You just declare the field on the embedded_schema.
embedded_schema do
field :title
field :author
end
Absinthe
One last problem was a wide-spread runtime error in the graphQL types. Previously a type of `:integer` would return a value like `10.4` without any trouble. But with the latest dependencies this would throw an error because of the type mismatch.
object :user do
field :some_avg, :integer
end
You can decide what type to use here for the specific domain you are working in. For simple averages it was a toss up between using string, to ensure accuracy, and float. Either way, the solution is easy enough - just change the type in the Absinthe type declaration.
object :user do
field :some_avg, :float
end
Thank You!
Thanks for reading and I hope someone else can move quickly as a result of my sharing this. Huge thanks to the community and core teams for making Elixir a great ecosystem!
