Home › Code Snippets

Return Input Number to Arbitrary Precision

Posted November 20th, 2010 by jtigor
in
Average rating
(0 votes)

Two functions to take an input number and return it with a reduced precision. First function does this "by hand" second uses APMath function. Motivation for this function was to enable getting a simple increment for a set variable control from an arbitrary input, which is unlikely to be an integer. Note that precision follows APMath convention that precision is number of decimal places rather than number of digits. 

 Function ChangePrecision(vNum, vPrecision, vPreserveSign)
	Variable vNum		//input number to operate on
	Variable vPrecision	//output precision (number of decimal places)
	Variable vPreserveSign	//1 == retain sign of input
 
	Variable vReturn
	Variable vExponent
	Variable vMantissa
	Variable vPwr
	Variable vSign
 
//this will cause an error with the log fcn
	if(vNum == 0)
		return 0
	endif
 
//precision less than 0 makes no sense
	if(vPrecision < 0)
		return vNum
	endif
 
//store sign of input and ensure input is positive 
	vSign = sign(vNum)
	vNum = abs(vNum)
 
//get mantissa of number	
	vExponent = floor(log(vNum))
	vPwr =  (1 *10^vExponent)
	vMantissa = vNum / vPwr
 
//change mantissa to desired precision	
	vMantissa = (round(vMantissa * 10^vPrecision)) / 10^vPrecision
	vReturn = (vMantissa) * vPwr
 
//if requested, change to original sign
	if(vPreserveSign == 1)
		vReturn *= vSign
	endif
 
//	Print log(vNum),vExponent, vmantissa, vPwr, vSign
 
	return vReturn
End
 
Function ArbPrec(vNum, vPrecision, vPreserveSign)
	Variable vNum		//input number to operate on
	Variable vPrecision	//output precision (number of decimal places)
	Variable vPreserveSign	//1 == retain sign of input
 
	Variable vSign
	Variable vOutput
	String sNum
 
//precision less than 0 makes no sense
	if(vPrecision < 0)
		return vNum
	endif
 
//store sign of input and ensure input is positive 
	vSign = sign(vNum)
	vNum *= vSign //same as abs(vNum)
 
//change to desired precision	
	APMath/N=(vPrecision) sNum = vNum
	vOutput = str2num(sNum)
 
//if requested, change to original sign
	if(vPreserveSign == 1)
		vOutput *= vSign
	endif
 
	return vOutput
end

Back to top