Does (FTP with) SSH2 always upload as binary? - GoFuckYourself.com

HQ · 10-18-2002, 06:31 AM

Does (FTP with) SSH2 always upload in Binary mode? ...or can you get it to upload via ASCII mode?

Every file I upload on my new server (protected with SSH2 security) is uploaded with binary mode. This means that all my DOS newlines are not being converted to Unix newlines. This fucks up all my scripts. (BTW, I am using Cute FTP Pro.)

Thanks for ANY help.

VooMan · 10-18-2002, 07:03 AM

I've never used cute ftp, but I've got Secure FX and that allows me to choose binary or ascii based on the file extension for ftp over SSH2...

http://www.vandyke.com/

Hope that helps. :-)

vik · 10-18-2002, 07:10 AM

I'm not sure about Cute FTP, but I'm using FTP Pro. It has 3 buttons in the middle of the two directories, the bottom button is a toggle switch to change between, binary, ascii, or auto. When I'm uploading script it is important for me to make sure I toggle it to ascii, as it sometimes defaults to binary (yes, I have my defaults set for ascii on .txt, .cgi, .pm, etc).

Check your program "help" files, they may shed some light on this for you.

vik

FreeNetPass Dave · 10-18-2002, 07:20 AM

On securefx (www.vandyke.com, the best one IMO), you can upload in either format.

DarkJedi · 10-18-2002, 07:35 AM

get FTP Voyager - it kicks serious ass

HQ · 10-18-2002, 08:24 AM

Quote:

Originally posted by VooMan
I've never used cute ftp, but I've got Secure FX and that allows me to choose binary or ascii based on the file extension for ftp over SSH2...

http://www.vandyke.com/

Hope that helps. :-)

Then why is my host saying "I'm fairly certain that the tools that utilize SSH and SSH2, like SCP and SFTP, are just binary." This statement is incorrect if you are certain that you are using SSH2 and you have the ability to use ASCII mode. Are you sure of this?

VooMan · 10-18-2002, 09:15 AM

Hi HQ,

I'm quite about Secure FX. The first time you upload a file it asks you how you want to send it, binary or ascii. You select the method, and you can also select to use that method for all files with that extension. Over time you've pretty much got all types of extensions taken care of. It doesn't matter if I've got a directory full of html, php, avi, and jpg mixed together, they all get sent the way I want them to without me having to worry about it.

Hope that helps.

CowboyAtHeart · 10-18-2002, 09:20 AM

Quote:

Originally posted by HQ
Does (FTP with) SSH2 always upload in Binary mode? ...or can you get it to upload via ASCII mode?

Every file I upload on my new server (protected with SSH2 security) is uploaded with binary mode. This means that all my DOS newlines are not being converted to Unix newlines. This fucks up all my scripts. (BTW, I am using Cute FTP Pro.)

Thanks for ANY help.

HQ: I'm not very familiar with windows ftp clients, although you should be able to set it, if not, the server will usually have a command dos2unix and unix2dos, which will fix them for you.
I use a linux workstation, so don't have much of a problem with things like this.

CowboyAtHeart · 10-18-2002, 09:23 AM

Quote:

Originally posted by HQ

Then why is my host saying "I'm fairly certain that the tools that utilize SSH and SSH2, like SCP and SFTP, are just binary." This statement is incorrect if you are certain that you are using SSH2 and you have the ability to use ASCII mode. Are you sure of this?

HQ: the conversion is done by the ftp server if you send it the command to go into ascii mode. FTP over ssh2, is just tunneling the ftp protocol over the ssh link. (same idea as http being tunneled over an ssl connection for https)
Other than the fact it's encrypted, the ftp server should see the data the same way any other does, it's decrypted before the ftpd actually see's it.

HQ · 10-18-2002, 09:58 AM

Quote:

Originally posted by VooMan
Hi HQ,

I'm quite about Secure FX. The first time you upload a file it asks you how you want to send it, binary or ascii. You select the method, and you can also select to use that method for all files with that extension. Over time you've pretty much got all types of extensions taken care of. It doesn't matter if I've got a directory full of html, php, avi, and jpg mixed together, they all get sent the way I want them to without me having to worry about it.

Hope that helps.

VooMan, Cute FTP Pro also has every one of those same options, but it does not work on my SSH2 connection. Just because those options exist, it does not mean they work for every type of conneciton. So are you sure you are on an SSH2 connection? Thanks!

HQ · 10-18-2002, 10:02 AM

Quote:

Originally posted by CowboyAtHeart
HQ: the conversion is done by the ftp server if you send it the command to go into ascii mode. FTP over ssh2, is just tunneling the ftp protocol over the ssh link. (same idea as http being tunneled over an ssl connection for https)
Other than the fact it's encrypted, the ftp server should see the data the same way any other does, it's decrypted before the ftpd actually see's it.

Thanks CowboyAtHeart. I was thinking it worked that way too. My CuteFTP Pro is working on my old server that use just plain FTP but does not work on my new server which is using SSH2 SFTP Triple DES. This 'bug' is quite annoying.

My host has suggested that I use a text editing program that saves in Unix newlines and then just upload everything as binary. I would rather not do this.

smashface · 10-18-2002, 10:07 AM

I dont know about cuteftp and its abilities. Whenever I get this I usually just login to the server and fix the files.

Some Unix experts may be able to write a script that you can invoke - what I do to the files with ^M is

vi foo

:%s/^M//g

:wq

This is a quick fix but not a solution

HQ · 10-18-2002, 10:11 AM

smashface, thanks, but there are plenty of ways to fix the files. "dos2unix" works but it changes the file permissions which makes it a pain in the ass. Anyway, these solutions are only temporary. Perhaps my settings in CuteFTP Pro itself are wrong. Should I be using Triple DES???

XXXManager · 10-18-2002, 11:58 AM

hey HQ
No - SSH ftp can be ASCII as well. You can select that in the menues probably.
Anyway - the problem you have dont necessarily have to do with ASCII/BIN. It has to do with the difference in CRLF between Unix and Windows

To fix it - (assuming you have perl on your machine) do the following after uploading the files
1. Stand in the directory you uploaded the files to
2. Assuming your files are .pl, run the two commands below:
2.1 find . -type f -regex '.*\.pl' | xargs perl -pi -e 's:\r\n:\n:g'
2.2 find . -type f -regex '.*\.pl' | xargs perl -pi -e 's:\r:\n:g'
This changes all the fuckups you get for moving a file from windows editors to unix/linux

Note: if they are not .pl but .php just change the extention of the find.

CAUTION: those commands go over ALL files it find recursively (=including sub-dirs). Make sure you indeed want to change all the files from the point you are at, downward.

I found no better way to do this. Just write a small shell script that you will execute every time you have an update to the scripts.

HQ · 10-18-2002, 01:04 PM

XXXManager, thanks for the fix, but this is not a sufficient solution. I just want to be able to code, upload, execute, repeat as I create my scripts without having to do all of this.

Also, how is my problem not ASCII/BIN? I understand that there are differences in newline characters, that's why FTP sites allow ASCII tranfers.

HQ · 10-18-2002, 03:44 PM

Riz · 10-18-2002, 05:32 PM

From the SSH server's perspective, yes, it is always binary. SFTP is not a tunnel over SSH of the standard FTP protocol. It is a seperate add-on module to sshd, comepletely distinct from standard FTP. It's just a similar looking interface with similar commands. As far as I know, sftp-server, the sshd module that implements the server side of SSH's ftp-like interface, is not capable of making the ASCII/Binary distinction.

As was said previously, standard ftp servers do implement the newline translation in ASCII mode (in addition to stripping the high bit) on the server side, so it works universally with all ftp clients.

With SSH, if the newline translation is to be done, it would have to be done in the client, which is why you are seeing some people reporting it works with some specific clients like SecureFX, and there may be others. I don't think there's any way to make it work if your client doesn't support it, however, short of using other scripts once the files are transfered (or getting windows versions of those scripts and doing it before the transfer) as others have suggested.

Hope that helps.

Riz.

HQ · 10-18-2002, 06:02 PM

Quote:

Originally posted by Riz
From the SSH server's perspective, yes, it is always binary. SFTP is not a tunnel over SSH of the standard FTP protocol. It is a seperate add-on module to sshd, comepletely distinct from standard FTP. It's just a similar looking interface with similar commands. As far as I know, sftp-server, the sshd module that implements the server side of SSH's ftp-like interface, is not capable of making the ASCII/Binary distinction.

As was said previously, standard ftp servers do implement the newline translation in ASCII mode (in addition to stripping the high bit) on the server side, so it works universally with all ftp clients.

With SSH, if the newline translation is to be done, it would have to be done in the client, which is why you are seeing some people reporting it works with some specific clients like SecureFX, and there may be others. I don't think there's any way to make it work if your client doesn't support it, however, short of using other scripts once the files are transfered (or getting windows versions of those scripts and doing it before the transfer) as others have suggested.

Hope that helps.

Riz.

Riz, very imformative first post. Thanks.

So you are basically saying that I need a text editor that saves the file in Unix format (with Unix newlines) ...OR... have a DOS-based text editor and then after the file is uploaded to the Unix server THEN convert it with a program on the server. Right?

XXXManager · 10-18-2002, 06:15 PM

Quote:

Originally posted by HQ

Riz, very imformative first post. Thanks.
So you are basically saying that I need a text editor that saves the file in Unix format (with Unix newlines) ...OR... have a DOS-based text editor and then after the file is uploaded to the Unix server THEN convert it with a program on the server. Right?

Yep, can do that as well ;)
I know a very nice editor that let you use unix format.
Its called Programmer-Studio and its coool. I love it. Lets you manage project trees and stuff, much better than anything else I have used - and trust me, I've used EVERYTHING

regardless of that - just an idea.. I have a "fix" script that I use whenever I upload files to a server. This script contains general fixes, like CRLF and other replacements AS WELL as many server specific fixes. Its a great way to port code from server to server..
(just as an example, I code URLs with http://localhost/... and fix replaces it with xxxmanager.com on the production servers or dev.xxxmanager.com onb the development server. It saves me aLOT of time and is very easy to use, I just type ./fix

Riz · 10-18-2002, 06:17 PM

Quote:

Originally posted by HQ

So you are basically saying that I need a text editor that saves the file in Unix format (with Unix newlines) ...OR... have a DOS-based text editor and then after the file is uploaded to the Unix server THEN convert it with a program on the server. Right?

Right. Or alternatively, get Windows/DOS versions of the conversion utilities and do the conversion before you upload the file.

A couple of examples would be:

http://www.simtel.net/pub/pd/51438.html
http://www.simtel.net/pub/pd/51439.html

Probably more on this page:

http://www.simtel.net/pub/msdos/txtutl/

if you're not happy with either of those.

Riz.

VooMan · 10-18-2002, 08:34 PM

Quote:

Originally posted by HQ

VooMan, Cute FTP Pro also has every one of those same options, but it does not work on my SSH2 connection. Just because those options exist, it does not mean they work for every type of conneciton. So are you sure you are on an SSH2 connection? Thanks!

Hi HQ,

I'm sure I'm using FTP over SSH2. However, not being a techie myself I have to admit that this discussion has started to go WAY over my head, so if there's something I'm missing I truly apologize. I know it works for me, but that's all I know at this point...

I believe there's a free trial for Secure FX at http://www.vandyke.com so give it a shot and see if it works for you. If it does, let me know. If not, I apologize for leading you down the wrong path.

10-18-2002, 06:31 AM	#1
HQ Confirmed User Join Date: Jan 2001 Posts: 3,539	Does (FTP with) SSH2 always upload as binary? Does (FTP with) SSH2 always upload in Binary mode? ...or can you get it to upload via ASCII mode? Every file I upload on my new server (protected with SSH2 security) is uploaded with binary mode. This means that all my DOS newlines are not being converted to Unix newlines. This fucks up all my scripts. (BTW, I am using Cute FTP Pro.) Thanks for ANY help.

10-18-2002, 07:03 AM	#2
VooMan Registered User Join Date: Jun 2002 Posts: 65	I've never used cute ftp, but I've got Secure FX and that allows me to choose binary or ascii based on the file extension for ftp over SSH2... http://www.vandyke.com/ Hope that helps. :-) __________________

10-18-2002, 07:10 AM	#3
vik Confirmed User Join Date: Oct 2002 Location: Minneapolis Posts: 144	I'm not sure about Cute FTP, but I'm using FTP Pro. It has 3 buttons in the middle of the two directories, the bottom button is a toggle switch to change between, binary, ascii, or auto. When I'm uploading script it is important for me to make sure I toggle it to ascii, as it sometimes defaults to binary (yes, I have my defaults set for ascii on .txt, .cgi, .pm, etc). Check your program "help" files, they may shed some light on this for you. vik __________________ PhotosByVik.com Photonudity.com

10-18-2002, 07:20 AM	#4
FreeNetPass Dave Confirmed User Join Date: Jun 2001 Posts: 270	On securefx (www.vandyke.com, the best one IMO), you can upload in either format. __________________ ICQ: 102858186 FreeNetPass AVS : $15/FREE membership even if the surfer cancels! $17 recurring Premiums! Adult Insider : Webmaster resources, daily news, FREE stuff for posting. PornStarBucks : Jenna Jameson, Briana Banks, Jill Kelly, and now Tera Patrick!

10-18-2002, 09:15 AM	#7
VooMan Registered User Join Date: Jun 2002 Posts: 65	Hi HQ, I'm quite about Secure FX. The first time you upload a file it asks you how you want to send it, binary or ascii. You select the method, and you can also select to use that method for all files with that extension. Over time you've pretty much got all types of extensions taken care of. It doesn't matter if I've got a directory full of html, php, avi, and jpg mixed together, they all get sent the way I want them to without me having to worry about it. Hope that helps. __________________

10-18-2002, 07:35 AM	#5
DarkJedi No Refunds Issued. Industry Role: Join Date: Feb 2001 Location: GFY Posts: 28,300	get FTP Voyager - it kicks serious ass

10-18-2002, 10:07 AM	#12
smashface Confirmed User Industry Role: Join Date: Sep 2002 Location: San Francisco, California Posts: 991	I dont know about cuteftp and its abilities. Whenever I get this I usually just login to the server and fix the files. Some Unix experts may be able to write a script that you can invoke - what I do to the files with ^M is vi foo :%s/^M//g :wq This is a quick fix but not a solution

10-18-2002, 10:11 AM	#13
HQ Confirmed User Join Date: Jan 2001 Posts: 3,539	smashface, thanks, but there are plenty of ways to fix the files. "dos2unix" works but it changes the file permissions which makes it a pain in the ass. Anyway, these solutions are only temporary. Perhaps my settings in CuteFTP Pro itself are wrong. Should I be using Triple DES???

10-18-2002, 11:58 AM	#14
XXXManager So Fucking Banned Join Date: Mar 2002 Location: Far out in the uncharted backwaters of the unfashionable end of the Western Spiral arm of the Galaxy Posts: 893	hey HQ No - SSH ftp can be ASCII as well. You can select that in the menues probably. Anyway - the problem you have dont necessarily have to do with ASCII/BIN. It has to do with the difference in CRLF between Unix and Windows To fix it - (assuming you have perl on your machine) do the following after uploading the files 1. Stand in the directory you uploaded the files to 2. Assuming your files are .pl, run the two commands below: 2.1 find . -type f -regex '.\.pl' \| xargs perl -pi -e 's:\r\n:\n:g' 2.2 find . -type f -regex '.\.pl' \| xargs perl -pi -e 's:\r:\n:g' This changes all the fuckups you get for moving a file from windows editors to unix/linux Note: if they are not .pl but .php just change the extention of the find. CAUTION: those commands go over ALL files it find recursively (=including sub-dirs). Make sure you indeed want to change all the files from the point you are at, downward. I found no better way to do this. Just write a small shell script that you will execute every time you have an update to the scripts. Last edited by XXXManager; 10-18-2002 at 12:03 PM..

10-18-2002, 01:04 PM	#15
HQ Confirmed User Join Date: Jan 2001 Posts: 3,539	XXXManager, thanks for the fix, but this is not a sufficient solution. I just want to be able to code, upload, execute, repeat as I create my scripts without having to do all of this. Also, how is my problem not ASCII/BIN? I understand that there are differences in newline characters, that's why FTP sites allow ASCII tranfers.

10-18-2002, 03:44 PM	#16
HQ Confirmed User Join Date: Jan 2001 Posts: 3,539	bump

10-18-2002, 05:32 PM	#17
Riz Registered User Join Date: Oct 2002 Posts: 2	From the SSH server's perspective, yes, it is always binary. SFTP is not a tunnel over SSH of the standard FTP protocol. It is a seperate add-on module to sshd, comepletely distinct from standard FTP. It's just a similar looking interface with similar commands. As far as I know, sftp-server, the sshd module that implements the server side of SSH's ftp-like interface, is not capable of making the ASCII/Binary distinction. As was said previously, standard ftp servers do implement the newline translation in ASCII mode (in addition to stripping the high bit) on the server side, so it works universally with all ftp clients. With SSH, if the newline translation is to be done, it would have to be done in the client, which is why you are seeing some people reporting it works with some specific clients like SecureFX, and there may be others. I don't think there's any way to make it work if your client doesn't support it, however, short of using other scripts once the files are transfered (or getting windows versions of those scripts and doing it before the transfer) as others have suggested. Hope that helps. Riz.