What is a good way to send large data sets to a client through API requests?

Questions / Help
1

Hello guys,
What is a good way to send large data sets to a client through API requests?
like i want to send user_login_history log to mixpanel
example :
like I have 50k login history in my table and i have to send this much data to service that i am using for my
project.

please explain, and write code for the same

2

like how should i write paginator code so the 50k rows data can be sent in 1k iteration, sending 50 row in on iteration, please help me with code

3

Generally, if you show what you’ve tried people are more likely/able to give advice. This is not typically a code writing service :wink:.

Hint: search hexdocs.com/elixir for chunk

4 Likes
4

Show us what have you tried, please.

1 Like
5

i am trying to do like this

query = from(p in UserLoginLog, order_by: [asc: p.inserted_at, asc: p.id])
page = MyApp.Repo.paginate(query, cursor_fields: [:inserted_at, :id], limit: 50)

after this i will push each page data to mixpanel after Jason.encode(page1) i am thinking like this
if i am on right track than guide me after getting page how can i push each page data one by one, it is there i am confused, how pagination work

6

Have you looked at Repo.stream?

1 Like