https://linzeyan.github.io/categories/r/Recent content in R on RickyHugo -- gohugo.ioenThu, 20 Jul 2023 15:44:11 +0800https://linzeyan.github.io/posts/2023/20230720-how-to-deal-with-a-50gb-large-csv-file-in-r-language/Thu, 20 Jul 2023 15:44:11 +0800https://linzeyan.github.io/posts/2023/20230720-how-to-deal-with-a-50gb-large-csv-file-in-r-language/<ul> <li><a href="https://stackoverflow.com/questions/39678940/how-to-deal-with-a-50gb-large-csv-file-in-r-language" target="_blank" rel="noopener">How to deal with a 50GB large csv file in r language?</a></li> </ul> <h3 id="question">question</h3> <div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-r" data-lang="r"><span style="display:flex;"><span>all <span style="color:#f92672">&lt;-</span> <span style="color:#a6e22e">read.csv.ffdf</span>( </span></span><span style="display:flex;"><span> file<span style="color:#f92672">=</span><span style="color:#e6db74">&#34;&lt;path of large file&gt;&#34;</span>, </span></span><span style="display:flex;"><span> sep <span style="color:#f92672">=</span> <span style="color:#e6db74">&#34;,&#34;</span>, </span></span><span style="display:flex;"><span> header<span style="color:#f92672">=</span><span style="color:#66d9ef">TRUE</span>, </span></span><span style="display:flex;"><span> VERBOSE<span style="color:#f92672">=</span><span style="color:#66d9ef">TRUE</span>, </span></span><span style="display:flex;"><span> first.rows<span style="color:#f92672">=</span><span style="color:#ae81ff">10000</span>, </span></span><span style="display:flex;"><span> next.rows<span style="color:#f92672">=</span><span style="color:#ae81ff">50000</span>, </span></span><span style="display:flex;"><span> ) </span></span></code></pre></div><h3 id="answer">answer</h3> <div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-r" data-lang="r"><span style="display:flex;"><span><span style="color:#a6e22e">library</span>(sqldf) </span></span><span style="display:flex;"><span> </span></span><span style="display:flex;"><span>iris2 <span style="color:#f92672">&lt;-</span> <span style="color:#a6e22e">read.csv.sql</span>(<span style="color:#e6db74">&#34;iris.csv&#34;</span>, </span></span><span style="display:flex;"><span> sql <span style="color:#f92672">=</span> <span style="color:#e6db74">&#34;select * from file where Species = &#39;setosa&#39; &#34;</span>) </span></span></code></pre></div>