C node passing messages
Now back onto the payment gateway project (cashier) working towards a complete and solid elixir payment library.
Any help welcome
Just learning and thinking how to apply.
I don’t have a lot of (any) extra time right now, so I’ve just been going slowly through some of the online resources for the language on elixir-lang.org.
Currently rewriting one project from Laravel PHP to Phoenix Elixir <3
Still very much in learning mode but started using Elixir for text mining. At some point I want to get very familiar with Nerves Project to use on my stash of SBC’s (RPi’s, BBB, ChiP and probably a few I have forgotten about) so I can bring extra nodes on-line for additional processing power as required.
Some updates on my crawling system in Elixir now that I’m at the 90 day mark of production. (see What Elixir related stuff are you doing? for context)
- The “clocking” mechanism I put in place has had no issues so far. It has actually helped a lot in troubleshooting the live system since I can adjust the clock rate at runtime. This allows me to really watch state changes without stopping the system entirely as well as being able to pause the system in its tracks and inspect a specific process - I think it turned out real well with the way I am handling the processes this way, at least for my use-case where I have a lot of post-processing after scraping and around 40 or so state changes per job.
- I now have 12 distributed nodes running in production and will likely scale out to about 20 by April or May.
- My current numbers as of right now are 20,554,562 crawls with 23,754 data errors - “data errors” represent something that went wrong during the entire crawl/post-processing cycle but does not represent data loss or anything like that. These 23k items still sent a response - just one that was less than ideal. At the end of the day its well under our SLA.
- I don’t have hard numbers here, but I think I’ve only had two legitimate “crashes”. One was due to some poor logging I put in a portion of code that caused my message queue to get too large. I paused my system but got impatient and just killed it rather than wait for the logging message queue to burn down. The other time I added some new state transitions, managed to misspell them, and forgot to add a default fall through
handle_infofor a scenario without a match. Both of these situations were human error on my part and they happened in the middle of the day when I was working on the system - that is better than 4am on a Sunday.
I’ll never really be done with this system as it will continue to grow, but I’m at the point where I sleep at night not worrying about this portion of my tech stack at all. I think that is a plus. I still deal with issues on our old crawlers, but as I migrate them each over to Elixir, life is getting easier for sure.
At the end of the day, my goal was to get to the point where I had stability and resiliency in a world that is tough to achieve this (crawling/scraping). I think I’m moving in the right direction.
Financially, the Elixir system has cut costs for this particular type of crawl job compared to our legacy system in regards to the amount hardware used and throughput achieved. Aside from this the data quality has improved quite a bit. Elixir isn’t the real hero on the data quality, but it allowed me to write a lot more code around data validation in many less lines than another language due to things like pattern matching. (I was lazy and writing crappy code before )
Hopefully this inspires someone to give it a go if you find yourself maintaining a legacy system with a lot of moving parts like myself. Not saying Elixir will be a life saver for you, but in my case its been a positive one.
Start small and experiment is my advice.
Right now I’m thinking about small-but-real-world projects to play around with.
I (still) don’t have a lot of time, so getting my hands dirty seems like the best use of the time I do have…
This week I have been
- Watching videos from Elixir Conf 2016
- Bought the Programming Phoenix
- Trying to promote Elixir/Phoenix to friends (unsuccessful so far)
- Starting to write a new app using Phoenix while learning it
Just released version 0.2.5 of Drab - the Phoenix extension to access the browser directly from Elixir and eliminate JS/Ajax.
Since I wrote to the forum the last time (v 0.1.0) many exctiting changes has been made:
- modularity: based on popular React fans request (@vic) now you can choose which Drab modules you want to use, so you are not fixed to jquery anymore
- new Query functions, like cycling (
update(:text, set: ["One", "Two", "Three"], on: "#thebutton")) - @Eiji
- more callbacks, like
- own Store (similar to Plug Session, key/value store available in your functions)
- access to Plug Session values (read-only, we don’t want to mess with Phoenix core)
Right now I am reading “Elixir in Action”
This is next to me after Elixir & OTP book! Currently reading it! For now love it, because the books is with real life examples and the benefits of OTP!
I am at chapter 10 now. I don’t know why but both “The Little Elixir & Otp Guidebook” and “Elixir in Action” are getting scary as soon as they introduce the Supervisors. I find it strange because the Supervisors themselves are such a simple concept. Maybe it is because by the time they introduce them, the complexity of the app has gone higher and it does require more thinking to follow.
I am reading The Little Elixir & OTP Guidebook.
Started learning Phoenix through this udemy course: The Complete Elixir and Phoenix Bootcamp (Udemy/self-published)
Besides tracking new hex packages and browsing this forum, I’m currently working on a good approach for dynamically creating Phoenix channels with dynamic subsets of presences server-side to establish incoming call routing for my startup AskVira.
I’ve read (Programming Elixir and Programming Phoenix), made a presentation and presented it (Introduction to Phoenix and Elixir), and now I’m working on building a project to help with distributed web applications, creating open source packages in Elixir as I go. The first one of these is Breaker.
Oh, and I’m looking for an Elixir-related job