darkmarmot
We have some code that creates unique hashes on documents to handle duplicate checks.
It wasn’t working on nodes that rebooted (letting dupes through) and as a last resort I changed from SHA to phash2 and it works perfectly now. This seems really strange to me. Is there some kind of timestamp seed or something in the crypto SHA I’m not aware of?
Here’s the code change that fixed our issues:
hash =
:erlang.phash2(doc, 1_000_000)
Here’s the original that is not working across restarts:
hash =
:crypto.hash(:sha, :erlang.term_to_binary(doc)) |> Base.encode64(case: :lower)
Maybe term_to_binary or encode64… but… it just surprised me a lot…
Where Next?
Trending in Questions
Other Trending Topics
Latest on Elixir Forum
Sponsor Spotlight
Deploy Phoenix to your own VPS with PaaS ease: push-to-deploy, zero downtime, auto SSL, managed Postgres. 14-day free trial at potions.io
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #podcasts
- #javascript
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #blog-post
- #elixirconf-us
- #elixir-ls
- #ai
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #hex
- #security
- #metaprogramming
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Elixir, Erlang & BEAM conferences from Code Sync - join us in 2026!
Bringing learning, work and compliance together since 1987.
Producing high quality Elixir screencasts since 2017.
The deployment platform built for Elixir: PaaS ease, VPS control.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
Showing Posts 1 to 2- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
cmkarlsson
Sorry for a few follow-up questions. I am not even sure I can help here.
doc? Is it a binary string or an erlang term? Something else?sha hashindividually to see which one is not working.My bet is that it is
term_to_binary. I am not sure it is meant to always produce the same result, only that it can be decoded withbinary_to_term.phash2on the other hand is designed to always produce consistent result when run on different version of erlang and machine architectures.benwilson512
The collision chance for phash2 is gonna be a lot higher than a sha though, so I’d definitely recommend finding a way to use a sha.