<turbo-stream action="append" target="posts_list"><template>    <div class="postbit" id="314506" data-post-id="314506">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="janhendrik-rust" src="/assets/icons/user-9f439610.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  janhendrik-rust
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>If I turn on debug logging, I get the following in an iex session:</p>
<p>`iex&gt;  from(f in SlowGoing.Platform.FloatsTest, limit: 10000) |&gt; SlowGoing.Repo.all()<br>
[error] Tds.Protocol (<a class="hashtag-cooked" href="/tag/pid/2279" data-type="tag" data-slug="pid" data-id="2279" data-style-type="icon" data-icon="tag" rel="nofollow"><span class="hashtag-icon-placeholder"><svg class="fa d-icon d-icon-square-full svg-icon svg-node"><use href="#square-full"></use></svg></span><span>pid</span></a>&lt;0.450.0&gt;) disconnected: ** (DBConnection.ConnectionError) client <a class="hashtag-cooked" href="/tag/pid/2279" data-type="tag" data-slug="pid" data-id="2279" data-style-type="icon" data-icon="tag" rel="nofollow"><span class="hashtag-icon-placeholder"><svg class="fa d-icon d-icon-square-full svg-icon svg-node"><use href="#square-full"></use></svg></span><span>pid</span></a>&lt;0.513.0&gt; timed out because it queued and checked out the connection for longer than 5000ms<br>
<a class="hashtag-cooked" href="/tag/pid/2279" data-type="tag" data-slug="pid" data-id="2279" data-style-type="icon" data-icon="tag" rel="nofollow"><span class="hashtag-icon-placeholder"><svg class="fa d-icon d-icon-square-full svg-icon svg-node"><use href="#square-full"></use></svg></span><span>pid</span></a>&lt;0.513.0&gt; was at location:</p>
<pre><code>(tds 2.3.4) lib/tds/tokens.ex:547: Tds.Tokens.decode_row_columns/3
(tds 2.3.4) lib/tds/tokens.ex:199: Tds.Tokens.decode_row/2
(tds 2.3.4) lib/tds/tokens.ex:65: Tds.Tokens.decode_tokens/2
(tds 2.3.4) lib/tds/tokens.ex:72: Tds.Tokens.decode_tokens/2
(tds 2.3.4) lib/tds/messages.ex:153: Tds.Messages.parse/3
(tds 2.3.4) lib/tds/protocol.ex:483: Tds.Protocol.decode/2
(tds 2.3.4) lib/tds/protocol.ex:544: Tds.Protocol.send_query/2
(tds 2.3.4) lib/tds/protocol.ex:175: Tds.Protocol.handle_execute/4
(db_connection 2.6.0) lib/db_connection/holder.ex:354: DBConnection.Holder.holder_apply/4
(db_connection 2.6.0) lib/db_connection.ex:1512: DBConnection.run_execute/5
(db_connection 2.6.0) lib/db_connection.ex:1607: DBConnection.run/6
(db_connection 2.6.0) lib/db_connection.ex:743: DBConnection.parsed_prepare_execute/5
(db_connection 2.6.0) lib/db_connection.ex:735: DBConnection.prepare_execute/4
(ecto_sql 3.11.1) lib/ecto/adapters/tds/connection.ex:52: Ecto.Adapters.Tds.Connection.execute/4
(ecto_sql 3.11.1) lib/ecto/adapters/sql.ex:996: Ecto.Adapters.SQL.execute!/5
(ecto_sql 3.11.1) lib/ecto/adapters/sql.ex:952: Ecto.Adapters.SQL.execute/6
(ecto 3.11.1) lib/ecto/repo/queryable.ex:232: Ecto.Repo.Queryable.execute/4
(ecto 3.11.1) lib/ecto/repo/queryable.ex:19: Ecto.Repo.Queryable.all/3
(elixir 1.14.3) src/elixir.erl:309: anonymous fn/4 in :elixir.eval_external_handler/1
(stdlib 4.2) erl_eval.erl:748: :erl_eval.do_apply/7`
</code></pre>
<p><code>[debug] QUERY OK source="floats_test" db=46034.3ms decode=0.3ms queue=1.1ms idle=1936.2ms SELECT TOP(10000) f0.[id], f0.[float_eight], f0.[float_five], f0.[float_four], f0.[float_nine], f0.[float_one], f0.[float_seven], f0.[float_six], f0.[float_three], f0.[float_two], f0.[int_one], f0.[int_three], f0.[int_two], f0.[inserted_at], f0.[updated_at] FROM [floats_test] AS f0 [] ↳ anonymous fn/4 in :elixir.eval_external_handler/1, at: src/elixir.erl:309</code></p>
<p>Which seems to indicate that it spent 46 seconds in db, 2 seconds in idle and almost no time in decode and queue</p>
<p>If I’m reading this right  Tds.Tokens.decode_row_columns eventually calls Tds.Types.decode_data – and this is where I get completely lost.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="314506" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/ecto-connection-times-out-when-using-tds-adapter-to-query-mssql-database/60928/12">Post #11</a>
	                </div>
	            </div>
              <div id="likers-container-314506" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="314506"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #11"></div>
  </section>
</div>
    <div class="postbit" id="314509" data-post-id="314509">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="muelthe" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/muelthe/120/35411_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  muelthe
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>Is it worth considering the MSSQL product version? For example. are any of the Express limitations affecting this: <a href="https://learn.microsoft.com/en-us/sql/sql-server/editions-and-components-of-sql-server-2019?view=sql-server-ver16#Cross-BoxScaleLimits" class="inline-onebox" rel="noopener nofollow ugc">Editions and Supported Features of SQL Server 2019 - SQL Server | Microsoft Learn</a></p>
<p>We’re connecting to an Azure hosted MSSQL DB for one of our systems using TDS, albeit quite small - I think it’s ~8000+ rows, but off the top of my head I’m not sure what kind of reponse times we’re seeing.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="314509" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/ecto-connection-times-out-when-using-tds-adapter-to-query-mssql-database/60928/13">Post #12</a>
	                </div>
	            </div>
              <div id="likers-container-314509" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="314509"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #12"></div>
  </section>
</div>
    <div class="postbit" id="314510" data-post-id="314510">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="benwilson512" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/benwilson512/120/1457_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  benwilson512
                  </h3>
		          </div>
						
			          <div class="user-title">
									<span>Author of Craft GraphQL APIs in Elixir with Absinthe</span>
			          </div>
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>If you copy that exact query into another client do you get a faster response time? Does the database log anything? Finally, is there an explain equivalent for TDS?</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="314510" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/ecto-connection-times-out-when-using-tds-adapter-to-query-mssql-database/60928/14">Post #13</a>
	                </div>
	            </div>
              <div id="likers-container-314510" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="314510"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #13"></div>
  </section>
</div>
    <div class="postbit" id="314511" data-post-id="314511">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="al2o3cr" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/al2o3cr/120/3457_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  al2o3cr
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>I don’t know the TDS protocol well enough to be sure this is the issue, but one innocuous-seeming line in the type-conversion code is a performance black-hole:</p>
<p><a href="https://github.com/elixir-ecto/tds/blob/f15b86871c41d004345d20130e0db215f03dc78a/lib/tds/types.ex#L508" class="inline-onebox" rel="nofollow">tds/lib/tds/types.ex at f15b86871c41d004345d20130e0db215f03dc78a · elixir-ecto/tds · GitHub</a></p>
<p>Reattaching <code>data</code> to <code>tail</code> is potentially VERY expensive if <code>tail</code> is the next 10k rows!</p>
<p>I wrote a microbenchmark to demonstrate this:</p>
<pre data-code-wrap="elixir"><code class="lang-elixir">defmodule Bench do
  def from_tds(&lt;&lt;&gt;&gt;), do: nil
  def from_tds(&lt;&lt;size::unsigned-8, data::binary-size(size), tail::binary&gt;&gt;) do
    data = data &lt;&gt; tail
    &lt;&lt;val::little-float-size(64), _::binary&gt;&gt; = data
    {val, tail}
  end

  def shorter_version(&lt;&lt;&gt;&gt;), do: nil
  def shorter_version(&lt;&lt;size::unsigned-8, data::binary-size(size), tail::binary&gt;&gt;) do
    &lt;&lt;val::little-float-size(64)&gt;&gt; = data
    {val, tail}
  end

  def big_binary(size) do
    &lt;&lt;8, 0, 0, 0, 0, 0, 0, 0, 0&gt;&gt;
    |&gt; List.duplicate(size)
    |&gt; Enum.join()
  end

  def run_example(size, fun) do
    big_binary(size)
    |&gt; Stream.unfold(fun)
    |&gt; Stream.run()
  end
end

[100, 1000, 10_000, 100_000, 1_000_000]
|&gt; Enum.each(fn size -&gt;
  {from_tds_time, _} = :timer.tc(&amp;Bench.run_example/2, [size, &amp;Bench.from_tds/1])
  {shorter_version_time, _} = :timer.tc(&amp;Bench.run_example/2, [size, &amp;Bench.shorter_version/1])

  IO.puts("#{size}:\t#{from_tds_time}\t#{shorter_version_time}\t#{from_tds_time / shorter_version_time}")
end)
</code></pre>
<p>Here are the results from my local machine (Elixir 1.13, 2015 MBP):</p>
<pre data-code-wrap="elixir"><code class="lang-elixir">100:		6250		22		284.09090909090907
1000:		1583		147		10.768707482993197
10000:		28594		1791	15.965382467895031
100000:		37877514	28739	1317.983019590104
1000000:	(I got bored)
</code></pre>
<p>That seems to echo the way you’re seeing query times scale with size.</p>
<hr>
<p>As to fixing the issue, I don’t know the TDS protocol at all. Are there situations where <code>length</code> (from the column metadata) and <code>size</code> (from the binary stream) would disagree?</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="314511" data-batch-url="/posts/batch_likers">
                        4
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/ecto-connection-times-out-when-using-tds-adapter-to-query-mssql-database/60928/15">Post #14</a>
	                </div>
	            </div>
              <div id="likers-container-314511" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="314511"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #14"></div>
  </section>
</div>
    <div class="postbit" id="314633" data-post-id="314633">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="Onor.io" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/Onor.io/120/960_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  Onor.io
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>That’s a good point.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="314633" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/ecto-connection-times-out-when-using-tds-adapter-to-query-mssql-database/60928/16">Post #15</a>
	                </div>
	            </div>
              <div id="likers-container-314633" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="314633"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #15"></div>
  </section>
</div>
    <div class="postbit" id="315010" data-post-id="315010">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="janhendrik-rust" src="/assets/icons/user-9f439610.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  janhendrik-rust
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>When I profile the following Ecto query on the sample project i posted (Thanks <a class="mention" href="/u/mjaric" rel="nofollow">@mjaric</a> for showing me how) :</p>
<pre data-code-wrap="elixir"><code class="lang-elixir">from(f in FloatsTest, select: [f.float_one,f.float_two,f.float_three,f.float_four,f.float_five,f.float_six,f.float_seven,f.float_eight,f.float_nine,f.int_one,f.int_two,f.int_three], limit: 10000) |&gt; Repo.all()
</code></pre>
<p>I get the following profiling results with this mix command</p>
<p><code>mix profile.fprof -e SlowGoing.Run.fetch_all</code></p>
<pre data-code-wrap="elixir"><code class="lang-elixir">                                                                  CNT    ACC (ms)    OWN (ms)     
Total                                                          1322773   52977.343   51457.080     
:fprof.apply_start_stop/4                                            0   52977.343       0.036     
anonymous fn/0 in :elixir_compiler_1.__FILE__/1                      1   52977.307       0.011     
SlowGoing.Run.fetch_all/0                                            1   52977.296       0.029     
SlowGoing.Repo.all/1                                                 1   52977.256       0.004     
SlowGoing.Repo.all/2                                                 1   52977.252       0.023     
Ecto.Repo.Queryable.all/3                                            1   52977.049       0.022     
Ecto.Repo.Queryable.execute/4                                        1   52976.993       0.057     
Ecto.Adapters.Tds.execute/5                                          1   47053.187       0.007     
Ecto.Adapters.SQL.execute/6                                          1   47053.180       0.016     
Ecto.Adapters.SQL.execute!/5                                         1   47053.159       0.012     
Ecto.Adapters.SQL.sql_call/5                                         1   47053.147       0.030     
Ecto.Adapters.Tds.Connection.execute/4                               1   47053.085       0.028     
DBConnection.prepare_execute/4                                       1   47052.907       0.021     
DBConnection.parsed_prepare_execute/5                                1   47046.775       0.014     
DBConnection.run/6                                                   1   46895.986       0.020     
DBConnection.run_prepare_execute/5                                   1   46895.485       0.016     
DBConnection.Holder.handle/4                                         2   46895.321       0.013     
DBConnection.Holder.handle_or_cleanup/5                              2   46895.308       0.036     
DBConnection.Holder.holder_apply/4                                   2   46895.248       0.041     
Tds.Protocol.msg_send/2                                              3   46894.854       0.122     
DBConnection.run_execute/5                                           1   46890.259       0.014     
Tds.Protocol.handle_execute/4                                        1   46890.163       0.039     
Tds.Protocol.send_query/2                                            1   46889.206       0.010     
Tds.Protocol.decode/2                                                2   46870.585       0.036     
Tds.Messages.parse/3                                                 2   46870.453       0.048     
Tds.Tokens.decode_tokens/1                                           2   46715.996       0.010     
Tds.Tokens.decode_tokens/2                                       10009   46715.986     157.969     
Tds.Tokens.decode_row/2                                          10000   46551.343      90.867     
Tds.Tokens.decode_row_columns/2                                  10000   46458.639      54.273     
Tds.Tokens.decode_row_columns/3                                 130000   46403.798    1857.341     
Tds.Tokens.decode_row_column/2                                  120000   43247.636     717.061     
Tds.Types.decode_data/2                                         120001   42486.278   30996.609     
:garbage_collect                                                 68406   10206.122   10206.122     
:lists.member/2                                                  90079    8701.587    1029.836     
Enum.map/2                                                           5    5922.734       0.032     
Enum."-map/2-lists^map/1-0-"/2                                   20005    5922.702     287.541     
Ecto.Repo.Preloader.query/6                                          1    5685.277       0.010     
anonymous fn/3 in Ecto.Repo.Queryable.postprocessor/4            10000    5532.761      92.117     
Ecto.Repo.Queryable.process/4                                   130000    5438.703    1159.499     
Ecto.Repo.Queryable.process_args/4                               10000    5381.206      56.798     
Enum.map_reduce/3                                                10007    5323.765      56.391     
Enum."-map_reduce/3-lists^mapfoldl/2-0-"/3                      130007    5266.946    1753.240     
anonymous fn/4 in Ecto.Repo.Queryable.process_args/4            120000    3471.793     672.895     
Ecto.Type.adapter_load/3                                        120000    1663.026    1108.097     
:suspend                                                         10812    1520.263       0.000     
Enum.reverse/1                                                   10010    1252.745     105.045     
:lists.reverse/2                                                 10019    1142.267     128.622     
Ecto.Type.of_base_type?/2                                       120000     514.021     505.688     
Ecto.Repo.Assoc.query/4                                              1     237.447       0.007     
Enum.reduce/3                                                        9     155.168       0.056     
Enum."-reduce/3-lists^foldl/2-0-"/3                              10035     155.134     106.856     
DBConnection.decode/4                                                1     150.775       0.016     
DBConnection.Query.decode/3                                          1     150.743       0.011     
DBConnection.Query.Tds.Query.decode/3                                1     150.718       0.019     
DBConnection.Query.Tds.Query.do_decode/3                         10001     150.680     102.344     
anonymous fn/3 in Ecto.Repo.Queryable.preprocessor/3             10000      95.536      53.035     
anonymous fn/1 in DBConnection.Query.Tds.Query.decode/3          10000      46.726      40.455     
anonymous fn/2 in Tds.Messages.parse/3                           10002      45.736      44.056     
Ecto.Repo.Queryable.preprocess/4                                 10000      40.233      40.233     
Tds.Protocol.msg_recv/1                                              2      20.777       0.045     
Tds.Protocol.msg_recv/2                                            102      18.742       2.853     
:gen_tcp.recv/2                                                    102      14.366       1.225     
:inet_tcp.recv/2                                                   102      11.560       0.580     
:prim_inet.recv/2                                                  102      10.980       0.566     
:prim_inet.recv0/3                                                 102      10.414       1.052     
:prim_inet.async_recv/3                                            102       7.556       1.456

... the rest of the profiler output is available at (https://github.com/janhendrik-rust/slow_going/wiki/Result-of-selecting-10000-Rows-by-12-columns) ...

</code></pre>
<p>Most of the time is spent in  <code>Tds.Types.decode_data/2</code> probably as a result of issue <a class="mention" href="/u/al2o3cr" rel="nofollow">@al2o3cr</a> outlined above</p>
<p>and a fair amount of time is spent  in GC.</p>
<p>This issue is also related to the fact that <code>decode_data/2</code> is called for each column in the result set and each row, in this example case <code>decode_data/2</code> gets called 120_000 times, 12 colums x 10_000 rows - so the problem seems get progressively worse as you add columns.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="315010" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/ecto-connection-times-out-when-using-tds-adapter-to-query-mssql-database/60928/17">Post #16</a>
	                </div>
	            </div>
              <div id="likers-container-315010" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="315010"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #16"></div>
  </section>
</div>
    <div class="postbit" id="315042" data-post-id="315042">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="al2o3cr" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/al2o3cr/120/3457_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  al2o3cr
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>Can you perform a similar experiment but only selecting <em>integer</em> columns? The slow path in <code>decode_data</code> is only for floats, so it shouldn’t be encountered.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="315042" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/ecto-connection-times-out-when-using-tds-adapter-to-query-mssql-database/60928/18">Post #17</a>
	                </div>
	            </div>
              <div id="likers-container-315042" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="315042"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #17"></div>
  </section>
</div>
    <div class="postbit" id="315130" data-post-id="315130">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="janhendrik-rust" src="/assets/icons/user-9f439610.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  janhendrik-rust
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>That is exactly right, when I profile an integers only query</p>
<pre data-code-wrap="elixir"><code class="lang-elixir">from(f in FloatsTest, select: [f.int_one, f.int_two, f.int_three, f.int_four, f.int_five, f.int_six, f.int_seven, f.int_eight, f.int_nine], limit: 10000) |&gt; Repo.all()
</code></pre>
<p>The query returns in 7 seconds instead of 52 seconds</p>
<pre data-code-wrap="elixir"><code class="lang-elixir">                                                                   CNT    ACC (ms)    OWN (ms)     
Total                                                           913951    7469.975    7287.849     
:fprof.apply_start_stop/4                                            0    7469.975       0.032     
anonymous fn/0 in :elixir_compiler_1.__FILE__/1                      1    7469.943       0.008     
SlowGoing.Run.fetch_all_ints/0                                       1    7469.935       0.022     
SlowGoing.Repo.all/1                                                 1    7469.867       0.005     
SlowGoing.Repo.all/2                                                 1    7469.862       0.022     
Ecto.Repo.Queryable.all/3                                            1    7469.682       0.023     
Ecto.Repo.Queryable.execute/4                                        1    7469.627       0.065     
Enum.map/2                                                           5    4632.802       0.026     
Enum."-map/2-lists^map/1-0-"/2                                   20005    4632.776     286.250     
Ecto.Repo.Preloader.query/6                                          1    4388.941       0.012     
anonymous fn/3 in Ecto.Repo.Queryable.postprocessor/4            10000    4242.289      91.559     
Ecto.Repo.Queryable.process/4                                   100000    4148.128     881.984     
Ecto.Repo.Queryable.process_args/4                               10000    4091.856      56.506     
Enum.map_reduce/3                                                10007    4034.502      56.159     
Enum."-map_reduce/3-lists^mapfoldl/2-0-"/3                      100007    3977.243    1324.443     
Ecto.Adapters.Tds.execute/5                                          1    2835.755       0.006     
Ecto.Adapters.SQL.execute/6                                          1    2835.749       0.014     
Ecto.Adapters.SQL.execute!/5                                         1    2835.730       0.009     
Ecto.Adapters.SQL.sql_call/5                                         1    2835.721       0.031     
Ecto.Adapters.Tds.Connection.execute/4                               1    2835.655       0.024     
DBConnection.prepare_execute/4                                       1    2835.503       0.024     
DBConnection.parsed_prepare_execute/5                                1    2830.201       0.017     
DBConnection.run/6                                                   1    2690.386       0.023     
DBConnection.run_prepare_execute/5                                   1    2689.441       0.016     
DBConnection.Holder.handle/4                                         2    2689.278       0.013     
DBConnection.Holder.handle_or_cleanup/5                              2    2689.265       0.029     
DBConnection.Holder.holder_apply/4                                   2    2689.215       0.038     
Tds.Protocol.msg_send/2                                              3    2687.875       0.136     
DBConnection.run_execute/5                                           1    2685.452       0.014     
Tds.Protocol.handle_execute/4                                        1    2684.381       0.037     
Tds.Protocol.send_query/2                                            1    2682.710       0.008     
Tds.Protocol.decode/2                                                3    2671.407       0.046     
Tds.Messages.parse/3                                                 3    2671.237       0.071     
anonymous fn/4 in Ecto.Repo.Queryable.process_args/4             90000    2602.208     503.440     
Tds.Tokens.decode_tokens/1                                           3    2526.041       0.016     
Tds.Tokens.decode_tokens/2                                       10012    2526.025     141.649     
Tds.Tokens.decode_row/2                                          10000    2379.666      92.614     
Tds.Tokens.decode_row_columns/2                                  10000    2282.176      55.985     
Tds.Tokens.decode_row_columns/3                                 100000    2224.965    1056.610     
Ecto.Type.adapter_load/3                                         90000    1229.861     819.005     
Tds.Tokens.decode_row_column/2                                   90000    1004.601     505.789     
Tds.Types.decode_data/2                                          90001     465.631     404.840     
Ecto.Type.of_base_type?/2                                        90000     385.967     378.347     
Ecto.Repo.Assoc.query/4                                              1     243.853       0.006     
:suspend                                                           728     182.126       0.000     
Enum.reduce/3                                                       10     145.852       0.055     
Enum."-reduce/3-lists^foldl/2-0-"/3                              10038     145.817     100.787     
Enum.reverse/1                                                   10012     141.788      93.154     
DBConnection.decode/4                                                1     139.798       0.019     
DBConnection.Query.decode/3                                          1     139.760       0.014     
DBConnection.Query.Tds.Query.decode/3                                1     139.729       0.023     
DBConnection.Query.Tds.Query.do_decode/3                         10001     139.685      97.943     
anonymous fn/3 in Ecto.Repo.Queryable.preprocessor/3             10000      95.830      54.176     
:garbage_collect                                                   340      95.473      95.473     
:lists.reverse/2                                                 10019      45.129      43.750     
anonymous fn/2 in Tds.Messages.parse/3                           10004      43.369      40.733     
Ecto.Repo.Queryable.preprocess/4                                 10000      41.153      40.356     
anonymous fn/1 in DBConnection.Query.Tds.Query.decode/3          10000      40.575      39.629     
Tds.Protocol.msg_recv/1                                              3      13.435       0.071     
Tds.Protocol.msg_recv/2                                             76      11.236       1.577     
:gen_tcp.recv/2                                                     76      10.032       0.814     
:inet_tcp.recv/2                                                    76       8.064       0.407     
:prim_inet.recv/2                                                   76       7.657       0.385     
:prim_inet.recv0/3                                                  76       7.272       0.714     
DBConnection.log/4                                                   1       5.217       0.006     
DBConnection.log/5                                                   1       5.211       0.022     
DBConnection.log/2                                                   1       5.044       0.008

... the rest of the profiler output is available at (https://github.com/janhendrik-rust/slow_going/wiki/Result-of-selecting-10000-Rows-by-9-columns-of-ints) ...


</code></pre>
<p>So it is deffinately the floating point codepath that  has bad performance problems.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="315130" data-batch-url="/posts/batch_likers">
                        3
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/ecto-connection-times-out-when-using-tds-adapter-to-query-mssql-database/60928/19">Post #18</a>
	                </div>
	            </div>
              <div id="likers-container-315130" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="315130"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #18"></div>
  </section>
</div>
    <div class="postbit" id="315199" data-post-id="315199">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="mjaric" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/mjaric/120/7788_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  mjaric
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>The issue is at <a href="https://github.com/elixir-ecto/tds/blob/f15b86871c41d004345d20130e0db215f03dc78a/lib/tds/types.ex#L508" rel="noopener nofollow ugc">line 508 in Tds.Types module</a>, just comment it out and it should fly <img src="https://forum.elixirforum.com/images/emoji/apple/wink.png?v=15" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>
<pre data-code-wrap="elixir"><code class="lang-elixir">        data_type_code == @tds_data_type_floatn -&gt;
          data = data &lt;&gt; tail 
          len = length * 8
          &lt;&lt;val::little-float-size(len), _::binary&gt;&gt; = data
          val
</code></pre>
<p>should be</p>
<pre data-code-wrap="elixir"><code class="lang-elixir">        data_type_code == @tds_data_type_floatn -&gt;
          len = length * 8
          &lt;&lt;val::little-float-size(len)&gt;&gt; = data
          val
</code></pre> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="315199" data-batch-url="/posts/batch_likers">
                        3
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/ecto-connection-times-out-when-using-tds-adapter-to-query-mssql-database/60928/20">Post #19</a>
	                </div>
	            </div>
              <div id="likers-container-315199" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="315199"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-solved cat-solved" title="Marked as solution"></div>
  </section>
</div>
    <div class="postbit" id="315582" data-post-id="315582">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="janhendrik-rust" src="/assets/icons/user-9f439610.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  janhendrik-rust
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>Thanks SO MUCH <a class="mention" href="/u/mjaric" rel="nofollow">@mjaric</a> and <a class="mention" href="/u/al2o3cr" rel="nofollow">@al2o3cr</a>.</p>
<p>I’ve just tested this fix and selecting 10_000 rows of 9 floats went from 45.9 seconds down to 173ms, that is an awesome  260x speed improvement.</p>
<p>The GC also only kicks in 455 times instead of 68406 times.</p>
<p>These numbers are obviously specific to my tests on my machine and only relevant when querying ““<em>lots</em>”” of rows with floating point numbers. Regardless, the performace is still at least two orders of magnitude better than before this fix.</p>
<p>I’ve had a look at the types.ex file history and <span class="mention">@naag</span> removed the same tail append for tds_data_type_intn in 22cb46a.</p>
<p>In my tests the floats still come through with their full precision so I don’t see any negative effect on the returned dataset after commenting out line 508</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="315582" data-batch-url="/posts/batch_likers">
                        3
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/ecto-connection-times-out-when-using-tds-adapter-to-query-mssql-database/60928/21">Post #20</a>
	                </div>
	            </div>
              <div id="likers-container-315582" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="315582"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #20"></div>
  </section>
</div>
</template></turbo-stream><turbo-stream action="replace" target="load-more-container"><template><div id="load-more-container" class="load-more-container">
    <a class="load-more-button" data-turbo-stream="true" href="/topics/60928/load_more?page=3">Load more posts (1 remaining)</a>
</div></template></turbo-stream>