Mapping over ~250 structs seems slow

Questions / Help
,
1

I’m doing a .map over a list of Ecto structs and that seems to take quite a lot of time on Heroku. I’ve been using AppSignal to benchmark that particular code, and there is a strong correlation between the number of items in the list and the time it takes for the function to complete.

For users that have a list of 200-300 structs, it could take from 400ms to 800ms to complete, which seems a lot to me. Originally I was doing Enum.map, but subsequently converted to using Task.async_stream which I think resulted in a minor improvement, but it’s still within the range above.

The formatting function itself does a few Map.put operations to convert ISO date keys to particular timezones (using Timex) and returns the struct.

Does that seem slow or it is acceptance performance when mapping lots of items? Or is there something obvious that I’m missing that I could do to map the structs faster? Thanks!

Using Elixir 1.4.2 and Phoenix 1.3.0rc, single Performance-M dyno on Heroku (I think 8 cores and 2.5GB ram).

2

The key information is that you’re using timex - it’s that part that is slow. I’ve experienced it couple times already.

1 Like
3

Ah, I didn’t think about that! Did you stick to Timex for timezone conversion then? Could Calendar be faster at all?

4

We had all the times in UTC anyway, so we used the native elixir functions. I’m not sure about calendar.

5

If you don’t use UTC time with one of the 2 timezone libraries. You need to lock to an old version of tzdata v0.1.201605.

See issue https://github.com/lau/tzdata/issues/31

It’s because tzdata stores all the timezone information in ets. And lookup in ets are pretty slow if you want to do thousands of lookups.

The old version of tzdata compiles all the timezone data with macro’s so is pretty fast compares to it. Though needs at least 2 Gb of memory on the computer when it get’s compiled because of it.

Both calendar and timex use tzdata so they both have the same problem with non utc timezones.

6

I highly doubt that the issue is tzdata. I would consider trying to use Calendar first and see if it improved. A couple hundred ets lookups should still take somewhere around a millisecond if the data involved is reasonably small.

We’ve had issues with Timex being slow as well, switching to Calendar removed all issues.

2 Likes
7

@benwilson512 Thanks, that’s great to hear! I will give Calendar a go this or next weekend, and will report back when I get a chance. Looking at our code (which is pre-Elixir 1.3) we could easily refactor a lot of it to use the new native Date/Time types as well.

8

Since my last post we refactored some of the code and switched to Calendar for timezone conversion; wherever it made sense, we also migrated data from DateTime to just Date. The same endpoint now finishes under 30ms :v:

2 Likes