IndiffererentAccess - Adaptation of HashWithIndifferentAccess to Elixir Maps/Plug

There are open source phoenix apps out there… I started by looking at one or two of them to see what they were doing and didn’t find anything I felt was compelling (but only looked at 1 or 2 for 5-10 minutes each), but I guess you’re saying there aren’t any good open source phoenix apps to point to? that’s… disappointing… but I guess possible…

  • Structs are mostly enforced at compile time. They save you from a part of errors you’d otherwise get at runtime.

  • Structs blow up at runtime if you touch them the wrong way (as your colleague did). And they give you a very clear error message.

  • Structs convey the message that you really want a certain piece of data to be shaped this way or that way. A good programmer will stop and think “why do they disallow removing of keys from this thing?” and maybe go and ask their colleague in charge of that code – this fosters communication in the team which is always a good thing.

  • Structs will slap you on the wrists if you introduce a new field and make it mandatory (the @enforce_keys [:field_a, :field_b, ...] annotation just above the defstruct expression) and you’ll have an exact list of compile errors to work through, which accelerates your work (as opposed to getting paranoid and adding more and more tests which might not even catch the problems that can come into existence by introducing the new field).

The benefits are practical and numerous but mostly gravitate towards “you get errors earlier and thus can react before any production / customer / business data can be harmed”.

I liked what Elixir Outlaws had to say about the whole explicit/implicit dichotomy… you may be right that this is “implicitly” doing something but, well… if thats true aren’t all plug pipelines doing things implicitly? Are they all bad? What makes this implicit behavior bad (particularly if it just replaces all string keys with atom keys) and other “implicit” behavior ok? I think it’s all on a spectrum and part of an evolving set of community norms and standards, I admit this is pretty far outside them at the moment and quite likely it should stay there, but that’s a different question then whether this is fundamentally more implicit/bad than other kinds of transformations…

Have you checked’s GitHub repo?

That was the first one I went and looked at actually. They still had a number of strings in their controllers. (mostly references to “slug” at a quick second glance, but still…)

def show(conn, %{"photo" => %{"width" => w, "height" => h}, "lat" => lat, "lon" => lon} = params) do
  # Convert string param values to integers or floats and do whatever must be done with them.

I am not saying Changelog’s code is exemplary, don’t get me wrong. Some things in there I’d very definitely approach differently. I believe the advice you’re given above by others is simply discovering that it’s just best to convert outside [and potentially malicious] data to a safe format your internal modules can work with, as quickly as possible. That includes potentially converting it to structs (see the “Parameter object” refactoring pattern for details). The edges of your app have to be rock-solid and that might include some ugly code. But keep it contained in the controllers and nowhere else.

I’m not against structs, but some of the recommended usage of them feels to me like people who want to make Elixir into a statically typed language, or make it seem closer to one… which isn’t a bad thing or something I’m against, but it’s not fundamentally “correct” either… it probably can/does lead to lower defect rates, but it also can make some refactoring and/or features take longer to develop, and part of what I value in Elixir is how flexible and fast to iterate/change it is… and it’s a big part of what my employers seem to value in it too. In many parts of the application there’s tolerance for bugs if we get substantial velocity out of it while we’re under time pressure and/or experimenting with new features that are not yet fully baked in scope or shape. I value the ability to start off more flexible and gradually harden the system over time and as needed.

No. They all modify one and the same object (per user / request, not globally). Nothing surprising or implicit about that, it’s just that some open-source plugs and pipelines save you the trouble of doing a ton of otherwise mundane work by yourself. I do not call that implicit behaviour, I call it convenience and you can lift the veil of mystery there by inspecting Phoenix’s source code which isn’t that big. :wink:

Sorry, the real time convo here is getting a little unwieldy/confusing, but thank you for the input and thoughts! And I agree this is all definitely best handled at the boundaries whenever possible, or perhaps I should say whenever practical as its always possible. There’s a huge diversity of what actually happens though, and once you have enough code where that’s not happening, part of the question is how to iteratively change code that works but isn’t designed they way you’d like and safely, gradually improve its design. This came up as one far-fetched idea to explore along that path is all.

Hmm… so here’s an interesting question… why doesn’t Phoenix and/or Plug use a struct instead of a bare map for its params objects coming through plug? If it did, a) it might do something like this more naturally, and/or b) I could make something like this that only worked this way on those Params structs…

If you only do that, it’s a perfectly fine scenario. But introducing a module that has to choose between resolving a string key or an atom key and one having precedence over the other – that’s definitely implicit / surprising behaviour.

Care to elaborate? Static typing has a number of advantages. Any error you can push to compile time as opposed to runtime is one less sleepless night for you fixing bugs at 2:30 AM.

It very definitely does, not “probably”, at least in my and many others’ experience.

That’s nothing that much older languages like PHP cannot give you as well. You clearly see the writing on the wall that Elixir’s creators and/or community would like more compile time enforcement, I see no point in you contesting this or arguing against it. Ruby is also extremely dynamic.

This is a very telling sign that I’d never work with your employers. :smiley:

That’s exactly what Elixir is giving you and you seem to fight it here in this thread. You can use your map with indifferent access; nobody is going to come to you and torture you with electric needles for it. But you also have the option to gradually harden your system right now and you seem to refuse by arguing with me and the others here.

So I am a little confused. You seem to get exactly what you want with Elixir but now, faced with the choice, you don’t want to use it the way you said you would. Help me understand?

Isn’t it obvious? Plug.Conn is a struct and parameters have to be a bare map because every app has their own set of parameters incoming to their controllers.

Well, yes, that may have been a bit of a brainfart/rushed thought… I think I was trying to make a point about how you can’t always use structs perhaps if you have an unknown set of keys, and our app is very dynamic/there are quite a few varieties of user configurations and settings coming stored in Postgres jsonb columns with customization for each different client…

(EDIT- I guess I was thinking/could argue hypothetically it could have been/could be a struct with an underlying map that has access behavior to access that map, and thats another version of this library I’m still considering trying out, that does approximately that to make an IndifferentParams struct that stores a map as basically its only value and uses access behavior to access that with atoms and wouldn’t care whether the atom exists, would just to_string the atom to look up, or fall back on to_string if the atom key is nil or something)

I don’t mean to seem argumentative/I’m not sure how much of what you or others are saying I disagree with at all, but playing devil’s advocate is part of how I figure out what I really think, so thank you for indulging me/apologies if the writing has come across the wrong way, it’s always a challenge to manage/understand tone both in the writing and in the reading. I appreciate and probably agree with you and most others here, but I threw this out there knowing more or less what most of the reactions would be, but wanting to try and play out and see if I was surprised by any or the points I or others made in either direction.

The PR I referenced above I think makes the library “only” do this, and I’m inclined to dismiss that version of it a little less than the original/I may spend 20 minutes trying that version out in my app to see how current behavior/test-failure situation compares… but I think I somehow didn’t do a good enough job communicating that nothing about the initial version I linked to was remotely finalized, it doesn’t have a single test, it was just a concrete direction (incorporating someones non-rescue based check for atom existence) to think about/discuss/consider more-or-less automated ways to not have to rely on strings in params maps.

I think your initial message was quite clear but the people around here definitely get PTSD and nightmares by such patterns. :003:

As a feedback, ultimately I parsed two things from your comments here:

  • You are kind of afraid of big refactorings. My opinion: you should not be. Refactoring your code to be stricter and less surprising is always worth it. If you really don’t have the time now, do your backlog features and keep taking notes for when you eventually get around to the refactoring (also chat with your employer; you are not a factory worker, programming is a creative job). Don’t try to cheat your way out with minimal work; well-behaved code requires focused and exhausting effort.

  • You are questioning the benefits of stronger typing / compile time errors. Nobody can convince you otherwise if you feel strongly about it but have in mind that languages like OCaml and Rust make an entire class of bugs impossible by the mere virtue of your program compiling successfully. We don’t have that in Elixir and we’re trying to gain some of it any way we can. If you value the mega-quick-time-to-working-MVP more then you probably shouldn’t use Elixir or push your team to move away from it.

1 Like

:smile: Sorry I didn’t put a trigger warning on it!

(also note I edited above briefly while you were replying)

I both am and am-not afraid of larger refactoring in different senses; I have done large refactorings successfully, but I do like playing around with different approaches/alternatives and have occasionally had a refactoring that passed all the tests and code reviews and seemed like a pure refactor every step of the way and still caused a bug in production for one reason or another. I like to think of it as a healthy fear, it won’t stop me from doing it, but I have respect for working code also and awareness that no amount of tests can prove the absence of bugs! :wink: I think I push myself harder than my employer does, but there is certainly mutual agreement that though it’s OK to charge the tech debt credit card, there’s also need to pay it down afterwards to avoid excessive interest payments.

I don’t know that I question the benefits of static typing (Elixir is strongly typed) but I do recognize that it has tradeoffs. I’m very interested in @keathley 's upcoming design-by-contract project as a potential sweet spot for these trade-offs, and a lot of what he’s spoken about on Elixir Outlaw’s resonates with me… and I think he’s one of the few way-better-than-me programmers in the community I’ve also seen question the value and advantage of “structs everywhere” vs Bare maps, so I’d be curious to hear his thoughts here. (EDIT - I beleive he said Bleacher report, which has of course scaled elixir quite far and well, has only a handful of structs in their codebase, so it at least provides some high profile counter-evidence to the idea that structs are the answer)

I don’t know what to say on the last point, re: probably shouldn’t use Elixir, but I don’t understand it and don’t think you’re going to convince me not to use it :wink:

You can do that. I can bet $20 that you’ll spend more time developing it now compared to the time you’ll save down the line by using it, though.

I don’t think this debate exists at all or that anyone says “structs are the answer” universally. Not sure if you noticed but most people in ElixirForum are refugees from other languages and frameworks, and a part of them still work with those or other languages but not Elixir. I feel the average quality of the programmers around here is pretty high and most would simply tell you “use bare maps vs. structs using your best judgement” which, yeah, is not a concrete advice, but then again you haven’t given much details yourself.

For what it’s worth, I’m only enforcing structs on controller params with a well-known and mandatory shape like pagination or ordering clauses. I do however convert string-keyed maps to atom-keyed maps very quickly and do exhaustive pattern matching when working with them further down. Getting FunctionClauseError gives you pretty clear error messages and I always go the extra mile to make all my function heads – including private ones – as strict as possible.

1 Like

You might well win that bet if I do it! Or rather, you could never win the bet because we never get to try things both ways (and especially not both-ways order independently!). But if I do it, it will not be to try and save the time but just to have a potentially fun thing to hack on as a side project/something without the time or other contraints from work, even if it was inspired by work or (maybe, but probably not TBH) winds up getting used there. In any case, happy to have chatted and appreciate your time and feedback. Sorry if I misrepresented your or other opinions above, I guess I was reacting to @NobbZ 's comment that

which didn’t sit entirely right with me/was perhaps the most concrete alternative I saw presented.

I can’t talk for him but I believe he meant that “normalised data” is what I called above “data with a well-known and mandatory shape, like pagination and ordering clauses”.

Happy to oblige the chat.

1 Like

“Mr. Keathley declines comment on this matter, except that he wishes to no longer be involved in this nonsense.” :wink: :laughing:

… but as someone else that works at Bleacher Report, I can say that structs aren’t required for scalability or anything like that. They’re just maps at the end of the day. It’s nice to be able to document and enforce the structure of things, like Plug.Conn does, when a lot of people need to collaborate on the same data structure. For your app code that’s changing all the time, it’s probably just as easy to use a map and try to keep it as simple as possible, at least until the code stabilized and you know what the “right” fields are going to be.

1 Like