darkmarmot
Strange behavior with SHA vs phash2
We have some code that creates unique hashes on documents to handle duplicate checks.
It wasn’t working on nodes that rebooted (letting dupes through) and as a last resort I changed from SHA to phash2 and it works perfectly now. This seems really strange to me. Is there some kind of timestamp seed or something in the crypto SHA I’m not aware of?
Here’s the code change that fixed our issues:
hash =
:erlang.phash2(doc, 1_000_000)
Here’s the original that is not working across restarts:
hash =
:crypto.hash(:sha, :erlang.term_to_binary(doc)) |> Base.encode64(case: :lower)
Maybe term_to_binary or encode64… but… it just surprised me a lot…
Most Liked
cmkarlsson
Sorry for a few follow-up questions. I am not even sure I can help here.
- Can you consistently reproduce this, the same document always produce different results after reboot of a node?
- What is
doc? Is it a binary string or an erlang term? Something else? - Is the initial hash and hash done on rebooted node done on the same nodes?
- If on different nodes are they running on different server architecture?
- Are you able to try the different parts of the
sha hashindividually to see which one is not working.
My bet is that it is term_to_binary. I am not sure it is meant to always produce the same result, only that it can be decoded with binary_to_term.
phash2 on the other hand is designed to always produce consistent result when run on different version of erlang and machine architectures.
benwilson512
The collision chance for phash2 is gonna be a lot higher than a sha though, so I’d definitely recommend finding a way to use a sha.
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Real-time error tracking, performance insights, and observability for devs.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Real-time error tracking, performance insights, and observability for devs.
Practical resources that improve the lives of professional developers.
Develop your skills with books, videos, and courses.
Producing high quality Elixir screencasts since 2017.
Courses that'll move you from confusion to "Aha, now I get it!"