Slow compiling and 404's

Currently I have a large and constantly growing Elixir Phoenix project, and the code scanner or something I have a few issues with (not all the issues exist on smaller code bases either).

  • First is that when runnng dev, even with the codereloader and livereloader entirely disabled (code commented out from endpoint.ex) about 10% of the time when requesting the, say, app.js or app.css files it will 404. The server message I can get right quick as just F5’ing enough times causes it:
iex(1)> [debug] ** (Phoenix.Router.NoRouteError) no route found for GET /css/app.css (MyServer.Router)
    (my_server) web/router.ex:1: MyServer.Router.match_route/4
    (my_server) web/router.ex:1: MyServer.Router.do_call/2
    (my_server) lib/my_server/endpoint.ex:1: MyServer.Endpoint.phoenix_pipeline/1
    (my_server) lib/plug/debugger.ex:122: MyServer.Endpoint."call (overridable 3)"/2
    (my_server) lib/my_server/endpoint.ex:1: MyServer.Endpoint.call/2
    (plug) lib/plug/adapters/cowboy/handler.ex:15: Plug.Adapters.Cowboy.Handler.upgrade/4
    (cowboy) src/cowboy_protocol.erl:442: :cowboy_protocol.execute/4

Another F5 will usually load it fine (rarely does it 404 twice in a row, but it will sometimes, even trice). This issue I’ve yet to see happen in ‘prod’.

  • Second random issue I have are if I do use the code reloader (have it enabled in endpoint) then every-single-request is dead-slow. I’ve had to workaround it by making a /reload route with the codereloader plug to do the reloading otherwise loading a page takes over 10 seconds (12 on average) every request, on routes where the codereloader plug is not on then they resolve in <200ms (where ‘prod’ is <1ms). 10+ seconds on every page load is dreadfully slow. When I load a page on the codereloader plug pipline there is no output from the server logs at all until it completes. If I open :observer and watch the processes the CPU and IO usage both reach near max utililzation for a short period of time for codereloader plug pipelines. Memory usage grows by about 6 megs during one of those requests then back down, which though not much, seems quite large regardless. The file_server process has an utterly massive amount of reductions that happen at that time, well over 12000000 (not a mis-type of zero’s) in reduction, which is absolute magnitudes above the next highest, which ends up being the Code Reloader code at about 100000 (eating 6 megs of memory at the same time before it gets collected). Compared to the next highest process with a reduction count of 75099, which is :observer stuff itself.

Point 1 is the most irritating by far as I’ve yet to find a workaround like /reload for point 2.

Point 2 is, painful when it happens, I’m guessing I hit some specific bad case for the codereloader plug, it has just been getting worse and worse and worse over time until its current average of 12s per run.

This is on Windows sad to say. Is there something I’m missing to help Point 1, or is there someway to see wtf codereloader is scanning for point 2 with perhaps some way to restrict where it is looking?

EDIT0: A full compile (mix clean && time mix compile) shows this, so even normal compiling is faster than the codereloader plug…

real    0m9.963s
user    0m0.030s
sys     0m0.090s

EDIT1: And yes, node_modules and such is at root, everything in web and lib is code and assets.

EDIT2: Oh, and for note, setting code_reloader: false, in the config does fix Point 1, but then restarting the server just to load a new javascript file (or manually copy it) is blegh…

2 Likes

Please try Elixir master, it supposedly makes the situation better.

Another idea is for you to figure out how to make symlinks work on Windows or speed up the code. The relevant code is here: https://github.com/elixir-lang/elixir/blob/master/lib/mix/lib/mix/utils.ex#L295. Given your debugging results, I believe most of the time is spent on copying the assets from priv to the priv in _build. If that is sped up, you should see large differences.

If you compile Elixir from master and put it in your $PATH, you can change that file to try things out and run make mix and it should recompile Mix for you. The README has instructions for source compiling on Windows if you haven’t done that before.

2 Likes

About the symlinks, according to this: https://msdn.microsoft.com/en-us/library/windows/desktop/aa365680(v=vs.85).aspx symlinks should work on windows without problems And here’s a article how to do this: https://technet.microsoft.com/en-us/library/cc753194(v=ws.11).aspx
I can’t say for other systems, but I’m 100% sure this works on Windows 8 and 10 because i used it myself with small SSD to move some folders inside c:/windows folder to other partition, and it worked.

And erlang’s make_symlink seems to make symlinks on systems >= Vista http://erlang.org/doc/man/file.html#make_symlink-2

I’m not on a windows machine right now, and can’t check it, but if it’s not working maybe there is an error in Erlang’s implementation?

1 Like

It is important for :file.make_symlink/2 that not only the operating system understands symlinks but the underlying filesystem as well. So it will only work for NTFS, not for any FAT AFAIK.

Also, sometimes creation of symlinks fails if the folder is shared or if you are running your elixir through a posix-wrapper like cygwin or msys, at least this is my observation and also true for symlinks created directly via ln (in posix wrappers).

Right, those are all great points and indicate it could be better. Those running on Windows please try to investigate if we are using symlinks or not and how we could improve it if not. You don’t need a Phoenix project: running mix compile on any project will execute the lines I mentioned above. I will be glad to help in any way possible.

I’m on Windows 10, I deploy to Windows Server 2016, all Windows on and above Windows 7 (with partial support for Vista, ick) have symlink support.

However, and this is funky, symlinks work without issue if an account is a Guest profile, or is an admin profile, but admin profiles get a UAC question on the attempt (at which you can then whitelist). So perhaps some option to support symlinks on Windows with the caveat that you will get asked that until you whitelist it?

However, I’m unsure if the Erlang file API has the ability to make symlinks on windows, been a while since I’ve tried, but the ln program from cygwin/mingw/etc…etc… packages on Windows can so a fallback to that if it exists (although running that for a ton of files would not be fast), or perhaps a NIF?

Ah, hmm, I’ll give it a try again. :slight_smile:

No wrappers here, just the global GNU tools, I do use symlinks around the system so I know they work fine here.

Awesome, will see if I can switch to Elixir master today after experimenting with making symlinks in :file

Some testing with :file.make_symlink/2 shows an immediate {:error, :eperm}, which indicates it is not using the correct command that brings up the UAC prompt, however I should be able to override it by running it directly as Administrator, and indeed I was:

iex> :file.make_symlink("blerp", "blorp")
{:error, :eperm}

So honestly I’d say just try :file.make_symlink/2, if you get {:error, :eperm} back then print that it is falling back to file copying since the faster symlinks require an administrator command prompt, then fallback to copying. :slight_smile:

EDIT0: Could even make it an option in the config/dev.exs or so perhaps?

EDIT1: WhooHoo! All I did was change my server process to run in as administrator directly and suddenly my reload page generation times are dang-near-instant! And lo and behold:

$ l _build/dev/lib/my_server/
total 122K
# snip
lrwxrwxrwx 1 <user> 1049089  44 Nov  9 08:20 priv -> /c/Users/<user>/Projects/ccc_server/priv

It does indeed already work! That definitely needs to be documented somewhere, preferably on stdout when running in dev mode. ^.^

Or better yet, a simple NIF that gains the process administrative access via UAC on startup (perhaps via a config option). Eh? :slight_smile:

So that solves point 2, will be testing point 1 as well but I’d bet money that one is fixed as well considering I’d bet they had the same root cause. ^.^

EDIT2: Do note, running as admin in dev I consider ‘fine enough’, but I sure as hell would not in prod, thankfully prod builds releases and does not need them. ^.^

What does it entail to run as an admin on Windows? Could it be done programmatically?

Btw, did you have a chance to try Elixir master and see if it improved anything before finding the fix above?

Great job!

What I’m doing is right-clicking my shell file and choosing “Run as Administrator” instead of just “Run”, you can also right-click a file, goto properties, Compatibility tab, then choose “Run this program as an administrator” (and you’ll get the UAC prompt every time you launch automatically), and yep it can be done programatically (the request to the Windows C API will cause a UAC prompt as well), this would require a NIF though and will cause a UAC prompt on either every access (if only promoting that one access) or once in a program run (if elevating the whole program).

However, all of those cause the full-screen UAC prompt to appear in a non-automateable way. If you want to create symlinks without it then you can run the program as Guest (why Guest can? Who knows, normal users cannot unless they have the make symlink permission system-wide, with a UAC prompt, admin users can via a UAC prompt regardless). Or you can do it by creating a Task with Run with highest privileges set for ‘now’, and there are a few other more hackery ways. Hence why I propose just a message the first time the code reloader is run saying that ‘Unable to symlink due to lack of permissions. Please run as an administrator to have significantly faster code reloading abilities. This will work fine though slower otherwise.’ With perhaps a URL linking to why Windows is stupid. ^.^

Not yet, I only had a few minutes to play with it at the time when I figured that out. Will do so when I get time. ^.^

EDIT: Also seen no issues with point 1, so they both seem good now. Only last issue I have is just a brunch issue (as usual). ^.^

So you are starting your terminal as an administrator, so you only need to confirm things once, right?

Can I also start the shell as a guest? If so, wouldn’t that be preferable?

Thanks for all the answers!

Precisely.

Sadly no, Guest users in Windows have access to almost nothing, and almost none of the file system, and it is hard to give them access to anything as well as Windows likes to lock it down (even sandboxing it), and even with access they cannot do very much. Guest access in Windows is not like any form of access you know of in Linux. The reason Guest’s have symlink access apparently is just because they cannot even reference any files to a symlink that they cannot already access, unlike other users. Windows is weird. ^.^

Phoenix master will now warn if we cannot create symlinks:

[warn] Phoenix is unable to create symlinks. Phoenix’ code reloader will run considerably faster if symlinks are allowed. On Windows, such can be done by starting the shell with “Run as Administrator”.

1 Like