Reusing established connection to google datastore

Questions / Help
1

So i’m trying to tackle the performance issues while making api calls to google datastore. i am trying to reuse the connection. can i use put_session, get_session, assign functions of Plug.Conn module to do this.
are there any other ways or packages to do this.

2

Can You give more details on what You are trying to achieve? Usually API don’t have sessions, but exchanges tokens.

1 Like
3

trying to reuse the opened gcs connection instead of making a new connection in every function call.

4

We still need more details. How are you making these calls now? Can you show us some code? What HTTP client are you using? What performance issue are you running into?

1 Like
5

i am using GoogleApi.Storage.V1.Connection.new(“ACCESS_TOKEN”) to establish connection to google cloud store. generating access token is taking some time (~400 ms)(i use Goth.Token.for_scope()). i am using a package which uses Httpoison to make requests.
i am not sure whether GoogleApi.Storage.V1.Connection.new(“ACCESS_TOKEN”) has any physical requirement because i got the output as shown, where access token is the only field which is useful.

%Tesla.Client{
  adapter: nil,
  fun: nil,
  post: [],
  pre: [
    {Tesla.Middleware.Headers, :call,
     [
       [
         {"authorization",
          "Bearer ACCESS_TOKEN"}
       ]
     ]}
  ]
}
6

It’s taking 400ms every time you call Goth.Token.for_scope? That’s definitely weird, since Goth caches the token value if it isn’t expired. It does look like Goth’s cache is sub-optimally written since it just sticks the value in a GenServer instead of :ets, but that should only be a factor if you’re making thousands of for_scope calls from different processes at the same time.

1 Like
7

I did not implement any cache mechanism. i just implemented it yesterday using :ets and things are good.
is :ets fine or do you have any suggestions.

8

You shouldn’t need to implement a cache, Goth has a cache built in. What I’m trying to figure out is: Does calling Goth.Token.for_scope take 400ms for you every time? How are you measuring this?

2 Likes
9

i used :timer.tc function.
thanks, i didn’t know that goth does it automatically. should we call some function of goth to refresh the token in Genstore.

10

It also handles that automatically: https://hexdocs.pm/goth/Goth.Token.html

I am happy to help more but you’re not really providing any code for us to try to diagnose. For example, show how you’re using :timer.tc

1 Like
11

sorry, actually my wording is misleading. goth took ~400 ms for the first api call.

defmodule R do
   def e do
    {:ok, token} = Goth.Token.for_scope("https://www.googleapis.com/auth/cloud-platform")
    conn = GoogleApi.Storage.V1.Connection.new(token.token)
  end

end

i passed above function to :timer.tc.
how to access the updated token from TokenStore, because calling

{:ok, token} = Goth.Token.for_scope("https://www.googleapis.com/auth/cloud-platform")

again is taking times.

12

It’s always going to take extra time for the first API call after it starts so that it can get a token. How long do calls after the first one take?

2 Likes
13

immediate calls are quick(~6 ms)

14

Right, so we’re back to the original question: What performance issue are you having

3 Likes
15

sorry, the problem is actually with the GoogleApi.Storage.V1.Api.Objects.storage_objects_insert_simple function, i assumed that the delay was due to Goth.Token.get.for_scope by mistake

16

I’m really, really trying to help, but the minimal information you’re providing in each post makes that very difficult. Imagine you were asking me about a performance problem, and all I said was

the problem is actually with the GoogleApi.Storage.V1.Api.Objects.storage_objects_insert_simple function,

How could you possibly help me? You need to tell us everything you would need from someone else if you were trying to help them.

I need to know at least ALL of:

  1. How long is the operation taking
  2. Does it always take that long, if not, when does it take that long
  3. How many objects are you trying to insert?
  4. Describe the overall API usage pattern
  5. Show the code with surrounding context
4 Likes
17

so sorry man,

  1. the function on an average 2-3 seconds (local machine), used below code to measure time.
start = System.monotonic_time(:second)
GoogleApi.Storage.V1.Api.Objects.storage_objects_insert_simple(
           conn,
           BUCKET_NAME,
           "multipart",
           metadata,
           path #path of image file
         )
time_spent = System.monotonic_time(:second) - start
  1. and yes it always takes approximately same time (2-3 sec), even when uploading the same image immediately
  2. uploading one image along with metadata
  3. the upload image task is taking too long and obstructing overall performance, now i am sure that GoogleApi.Storage.V1.Api.Objects.storage_objects_insert_simple is the culprit
GoogleApi.Storage.V1.Api.Objects.storage_objects_insert_simple(
           conn,
           BUCKET_NAME,
           "multipart",
           metadata,
           path #path of image file
         )
18

Excellent, thank you. Cool, so the main question we need to figure out is if this has something to do with your internet and its connection to Google, or if it’s something else. Have you tried uploading the same file via google storage command line?

1 Like
19

no, i’ve considered internet as issue.
ill try in that direction and let you know

20

internet does have an impact but the function also is taking some time (~100 ms) when my internet speed is ~3 MBps.