<turbo-stream action="append" target="posts_list"><template>    <div class="postbit" id="272141" data-post-id="272141">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="AstonJ" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/AstonJ/120/15_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  AstonJ
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>So I’ve browsed a few more posts/articles about ML on Macs and the vibe I’m getting is that if you’re a Mac user and need CUDA, don’t bother <img src="https://forum.elixirforum.com/uploads/default/original/2X/7/776053aa22968f83c10e90c98b23aa0ddd5c9d47.gif?v=15" title=":lol:" class="emoji emoji-custom" alt=":lol:" loading="lazy" width="20" height="20"> instead:</p>
<ul>
<li>
<p>Use AWS or some cloud service (about $3 an hour).</p>
</li>
<li>
<p>Build a Linux workstation and just SSH into it.</p>
</li>
<li>
<p>Or if you want to use your Mac, get something like a Razer Core X and dual boot into Linux or windows (or run from an ssd) - and don’t get a high end GPU unless you really need it (wait for them to come down in price or see what happens in the space).</p>
</li>
</ul>
<p><em>Does that sound about right? Maybe things have moved along since?</em></p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="272141" data-batch-url="/posts/batch_likers">
                        4
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/which-gpus-can-you-use-with-nx-axon-etc/52458/12">Post #11</a>
	                </div>
	            </div>
              <div id="likers-container-272141" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="272141"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #11"></div>
  </section>
</div>
    <div class="postbit" id="279387" data-post-id="279387">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="ekhall" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/ekhall/120/29644_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  ekhall
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>I’ve done the same - I have a Fedora 37 box with a 1080 in it. I’ve used linux on and off since '96 and at the moment it’s finally feeling usable without a lot of tinkering.</p>
<p>My problem <em>now</em> is that Fedora 37 ships with CUDA 12 and everything else seems to  be stuck on 11.7/11.8. Sigh <img src="https://forum.elixirforum.com/images/emoji/apple/slight_smile.png?v=15" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> Maybe someday this will all work easily.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="279387" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/which-gpus-can-you-use-with-nx-axon-etc/52458/13">Post #12</a>
	                </div>
	            </div>
              <div id="likers-container-279387" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="279387"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #12"></div>
  </section>
</div>
    <div class="postbit" id="279536" data-post-id="279536">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="AstonJ" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/AstonJ/120/15_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  AstonJ
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>Awesome - perhaps if you get a minute you could write a blog post or guide to help others get set up?</p>
<p>Also what do you think you might use for production? (I wrote up some options <a href="https://forum.elixirforum.com/t/elixir-deployment-options-with-gpu/52697/4" rel="nofollow">here</a> :D)</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="279536" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/which-gpus-can-you-use-with-nx-axon-etc/52458/14">Post #13</a>
	                </div>
	            </div>
              <div id="likers-container-279536" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="279536"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #13"></div>
  </section>
</div>
    <div class="postbit" id="286777" data-post-id="286777">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="toranb" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/toranb/120/16117_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  toranb
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>I got this working with the RTX 3060 and found it was painless if you can install PopOS and the nvidia drivers + cuda 11.2. I also recorded a short benchmark and included the side by side of fine tuning with CPU /GPU to give people a sense of how much faster the feedback loop is when you bolt on this nvidia GPU</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://toranbillups.com/blog/archive/2023/04/29/training-axon-models-with-nvidia-gpus/">
  <header class="source">

      <a href="https://toranbillups.com/blog/archive/2023/04/29/training-axon-models-with-nvidia-gpus/" target="_blank" rel="noopener nofollow ugc">toranbillups.com</a>
  </header>

  <article class="onebox-body">
    

<h3><a href="https://toranbillups.com/blog/archive/2023/04/29/training-axon-models-with-nvidia-gpus/" target="_blank" rel="noopener nofollow ugc">Training Axon Models With Nvidia GPUs</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="286777" data-batch-url="/posts/batch_likers">
                        7
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/which-gpus-can-you-use-with-nx-axon-etc/52458/15">Post #14</a>
	                </div>
	            </div>
              <div id="likers-container-286777" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="286777"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #14"></div>
  </section>
</div>
    <div class="postbit" id="314995" data-post-id="314995">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="AndyL" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/AndyL/120/3659_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  AndyL
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>I installed an RTX 3060 in a <a href="https://www.amazon.com/dp/B085TH77RT?psc=1&amp;ref=ppx_yo2ov_dt_b_product_details" rel="noopener nofollow ugc">Thunderbolt eGPU case</a>.  Also upgraded my Ubuntu dev box to an 16-core i7 / 64GB ram and 4TB NVME drive.</p>
<p>I’ve come to believe that this is a minimal configuration for local ML development, for small training runs and to generate embeddings for prototype apps.</p>
<p><a href="https://github.com/Syllo/nvtop" rel="noopener nofollow ugc">Nvtop</a> has been super useful to verify GPU operation.    Besides Nx, <a href="https://ollama.ai/" rel="noopener nofollow ugc">Ollama</a> has been a great tool for experimentation and learning, and gives a restful interface for integration with Elixir apps.</p>
<p>To generate one embedding with my setup, it takes ~90 seconds on the CPU, and ~2.7 seconds on the GPU.</p>
<p>Now and then I exceed memory capacity of the <a href="https://www.amazon.com/MSI-GeForce-RTX-3060-12G/dp/B08WPRMVWB/ref=sr_1_1?dib=eyJ2IjoiMSJ9.ZeIodXESOr6kfykS7rxQAjgMDSI3dBF4CJkO2JBYsfzGb5TmAC7Iq-ssKbSx6dZPfydEqEEm8QbrEBzgIsuOSwMqUqSTdXqakN1s6o9efRRDKgu2h-HDm-w6wrdDeXy2TzDu543o2TBaAcuGooE72moOweDG_7QHn2p3ydVK3Qz9YNe4eL10JdH1xWCkaerEJbUctdxtFsqglyzJmEKAEbb_ZmFFz-RiNJMxd8iSzDI.A7_-hnFEP3VrPDiLDoimQKp-3t5Rvn8Brwfb0HA8kYY&amp;dib_tag=se&amp;keywords=rtx+3060&amp;sr=8-1&amp;ufe=app_do%3Aamzn1.fos.c3015c4a-46bb-44b9-81a4-dc28e6d374b3" rel="noopener nofollow ugc">3060 GPU</a> (12GB/~$300).  The workaround is to use smaller models, good enough for experimentation and prototyping.</p>
<p>The <a href="https://www.amazon.com/NVIDIA-RTX-3090-Founders-Graphics/dp/B08HR6ZBYJ/ref=sr_1_2?dib=eyJ2IjoiMSJ9.lGipulvDpvz5oJqWZUYQCGvoHOSMl9XN4_GIbmEt-MPdORdIt1fFfGmOV3i34ihyfBiXsFivF4mFav4r4f7UrssSsnhzYHAqfui1DLRGWusponx2CMikfyaIdP5VRg5G2oKmiUV0154KuEXrconQK56Rl9Ty3291bT7R8DB3-Y6seIzZzWIGfZE8aGIHYN6zUPgPSDpFdGcWMnuim2G40HgOGCTqC_YxKcI1ADgrCjE.JASk7zpylR8YMXNwlfy4BUF_4fCaK1s6ckiLqmqSDo4&amp;dib_tag=se&amp;keywords=rtx+3090&amp;sr=8-2&amp;ufe=app_do%3Aamzn1.fos.765d4786-5719-48b9-b588-eab9385652d5" rel="noopener nofollow ugc">3090 GPU</a> (24GB / ~$1500) has 2x more ram but 5x more expensive.</p>
<p>Beyond that <a href="http://Fly.io" rel="noopener nofollow ugc">Fly.io</a> has on-demand <a href="https://community.fly.io/t/fly-gpus-are-here/16110" rel="noopener nofollow ugc">Nvidia A100s</a> - 40GB @ $2.50/hr, 80GB @ $3.50/hr.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="314995" data-batch-url="/posts/batch_likers">
                        2
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/which-gpus-can-you-use-with-nx-axon-etc/52458/16">Post #15</a>
	                </div>
	            </div>
              <div id="likers-container-314995" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="314995"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-last-post cat-last-post" title="Last post!"></div>
  </section>
</div>
</template></turbo-stream><turbo-stream action="replace" target="load-more-container"><template><div id="load-more-container" class="load-more-container">
    <span class="all-loaded">— All posts loaded —</span>
</div></template></turbo-stream>